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(57) Abstract 

A basic transcoder, that decodes an encoded 
data stream and re-encodes it, may be used for 
locally modifying data in said stream. Said 
modification is carried out by defining a predicted 
data stream on the basis of a given modifying 
data stream and adding to the decoded stream 
- between the decoding and re-encoding parts 
of the transcoder - the diffence between the 
modifying data stream and the predicted one. In 
an improved implementation, the modifying data 
stream is preprocessed so that it may be obtained 
in the decoded data stream a local substitution of 
said modifying data to the decoded ones. 
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Method and device for modifying data in an encoded data stream. 



FIELD OF THE INVENTION 

The present invention relates to a method of modifying data in an encoded data 
stream corresponding to successive pictures divided into sub-pictures, comprising the steps of: 

- decoding said encoded data stream ; 
5 - re-encoding the decoded data stream. 

The invention also relates to a video processing device for carrying out said method. This 
invention, for instance useful when a broadcaster wants to introduce additional data such as its 
own logo into a sequence of pictures, finds applications not only in the field of MPEG-2 
compression (MPEG-2 will be the common standard for TV broadcasting), but more generally 
10. in any digital video data compression system. 

BACKGROUND OF THE INVENTION 

In such a situation, i.e when additional digital data have to be added before 
transmission to an existing coded bitstream, the simplest solution is to decode said bitstream 
1 5 before carrying out said addition. The modified bitstream is then re-coded and transmitted. 
Unfortunately, a full decoding can generally be considered as expensive, as it requires a 
decoder and an encoder. Moreover, re-encoding with re-estimated modes and vectors may 
introduce artefacts on the areas left untouched by the logo addition. 



20 SUMMARY OF THE INVENTION 

It is therefore a first object of the invention to propose a low-cost solution to 
add data in a previously existing coded bitstream. 

To this end, the invention relates to a method such as described in the opening 
paragraph of the description and which is moreover characterized in that it also comprises the 
25 steps of : 

- defining a predicted data stream on the basis of an input modifying data 

stream ; 

- adding to the decoded data stream, before re-encoding it, the difference 
between the modifying data stream and the predicted one. 
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This method has for effect to convert the additional data, thanks to the mixing 
step, into a specific correcting signal that can then be added to the residue signal of the 
transcoder. 

Another object of the invention is to propose an improved method for carrying 
5 out an insertion of data with an improved quality with respect to said addition of data. 

To this end the invention relates to a method characterized in that it also 
comprises the step of preprocessing the modifying data stream in order to obtain in the 
decoded bitstream a substitution of said modifying data to the decoded ones. 

Another object of the invention is also to propose devices for implementing the 
1 0 above-mentioned methods. 

To this end the invention relates, in a first implementation, to a device for 
adding data to a video coded data stream, characterized in that it comprises : 

(1) a transcoder, for decoding said data stream and re-encoding it ; 

(2) an addition branch, for defining : 

1 5 (i) on the basis for these additional data and motion vectors defined in 

the transcoder, a predicted signal ; 

(ii) the difference between said additional data and said predicted 
signal, in view of an addition of said difference into said data stream between decoding and re- 
encoding parts of said transcoder. 

20 The invention also relates, in a second implementation, to a device for inserting 

data into a video coded data stream, characterized in that it comprises : 

(1) a transcoder, for decoding said bitstream and re-encoding it ; 

(2) an insertion branch, for defining : 

(i) on the basis of these additional data a preprocessed signal ; 
25 (ii) on the basis of said preprocessed signal and motion vectors defined 

in the transcoder, a predicted signal ; 

(iii) the difference between said preprocessed additional data and said 
predicted signal, in view of an insertion of said difference into said bitstream between 
decoding and re-encoding parts of said transcoder. 

30 In the US patent N° 5691986, a method and apparatus for inserting data into an 

encoded data stream are described. However, in the situation as described, said stream is in 
fact a multiplex of elementary data streams that have to be demultiplexed in order to extract 
one of these elementary streams, to perform a data reduction operation on this extracted 
stream, and to insert said data into the reduced data stream. The present invention has a 
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distinct object, which is either the addition of data in a picture or, in an improved embodiment, 
. the local replacement of a part of a picture by other picture data. 

BRIEF DESCRIPTION OF THE DRAWINGS 
5 These and other aspects of the invention will be apparent from and elucidated 

with reference to the embodiment described hereinafter. 
In the drawings : 

- Figs.l and 2 show conventional video decoder and encoder ; 

- Fig.3 shows a possible scheme of a transmission chain for adding a log to an 
1 0 incoming bitstream ; 

- Fig.4 illustrates the outline of a known transcoder, without logo adder ; 

- Fig. 5 illustrates such a transcoder when it is, according to the invention, 
provided with a logo adder ; 

- Fig.6 also illustrates such a transcoder, but when it is, according to the 
1 5 invention, provided with a logo insertion device ; 

- Fig.7 illustrates a computation made in said logo insertion device ; 

- Figs. 8 to 10 illustrate the processing carried out on the luminance and 
chrominance components, and Figs. 1 1 and 12 a logo clipping carried out after said 
processings. 

20 

DESCRIPTION OF THE INVENTION 

A conventional video decoder such as shown in Fig.l includes a decoding 
channel 12, which comprises in cascade a variable length decoding circuit 1, an inverse 
quantizing circuit 2 and an inverse frequency transform circuit 3 (respectively : VLD, IQ, 

25 IDCT), and a motion compensation channel 14, which comprises a picture memory 4 
receiving the output signals of the decoder, a motion compensation circuit 5 (said 
compensation taking into account the output signals of this memory 4 and motion vectors V(n) 
received by the decoder) and an adder 6 of the output signals of circuits 3 and 5 (respectively : 
MEM, COMP, A). The output picture of the decoder (also sent to the picture memory 4) is 

30 reconstructed by adding the prediction (output of circuit 5) to the decoded residual signal 
available at the output of the circuit 3 of the decoding channel 12. 

A conventional video encoder such as shown in Fig.2 includes an encoding and 
decoding channel 13, which comprises a discrete cosine transform circuit 25, a quantizing 
circuit 26, a variable length coding circuit 27 (respectively : DCT, Q, VLC) and, at the output 
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of the circuit 26, in cascade, an inverse quantizing circuit 28 and an inverse discrete cosine 
transform circuit 29 (respectively : IQ and IDCT), and a prediction channel 1 1, allowing to 
subtract a motion compensated prediction from the input signal and which comprises an adder 
21, for the reconstruction of the pictures before prediction, a picture memory 22, a motion 
5 compensation circuit 23 and a subtracter 24 (respectively : A, MEM, COMP, S). Said 
compensation takes into account previously estimated motion vectors V(n). 

A possible method and a device allowing to add data such as a logo to an 
incoming bitstream in a transmission chain are illustrated in Fig.3. Said chain comprises a first 
encoder 31 (called "encoder 1"), a sub-system 305 for the addition of the logo to the coded 

10 bitstream available at the output of the encoder 31, and, after transmission, a decoder 35 

("decoder 2"). The sub-system provided between said encoder and decoder itself comprises, as 
illustrated, a decoder 32 ("decoder 1"), a logo adder 33 and an encoder 34 ("encoder 2"). Then, 
starting from said decoder 32 and said encoder 34 and taking advantage of their 
complementarity, some simplifications will be made, in order to finally reach the outline of the 

1 5 transcoder according to the invention. 

According to Fig.2, one has indeed, for the first encoder 3 1 : 
R(n,l) = I(n) - P(F(n-l), l ; V(n)) (1) 
where the index (,1) in R(.) and P(.) designates the "first" encoder 31 (= encoder 1), I(n) is the 
original video input of the encoder, P(F(n-l), 1 ; V(n)) is the prediction signal computed by 

20 applying motion vector V(n) to the previously "decoded" picture F(n-l) and which has to be 
subtracted from the original input bitstream I(n) in order to obtain the residual signal R(n) to 
be coded, and R(n) designates this residual signal. It must be noted that the signal R s (n) 
available at the input of the prediction channel of the encoder differs from R(n) by a value e(n) 
called the coding error ; R'(n) being equal to R(n) + e(n), the signal at the output of the adder 

25 of this prediction channel is therefore F(n) = I(n) + e(n). 

For the first decoder 32 that follows the encoder 31, one has similarly, 
according to Fig. 1 : 

F(n,l) = R'(n,l) + P(F(n-l), 1 ; V(n)) (2) 
where the index (,1) in F(.), R'O and P(.) designates the "first" decoder 32, R'(n) is the 
30 decoded residual signal, P(F(n-l), 1 ; V(n)) is the prediction to be added to R'(ii), and I'(n) 
designates the output of the decoder. As seen above, F(n,l) may also be written in the form : 

F(n,l) = I(n) + e(n,l) 
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that is to say, the output of the decoder is the sum of the original input signal I(n) and of the 
coding error e(n) during the coding operation of I(n). At the output of the logo adder 33, one 
has consequently : 

J' (n, 1 )=r(n, 1 )+Logo(n) (3) 
5 where F(n) is the output of the decoder 32 and Logo(n) the data (for instance, a logo) to be 
added to the main bitstream. The resulting output J'(ri) is sent towards the encoder 34. 

For this second encoder 34, one has (similarly to the previous case of the first 

encoder 31) : 

R(n,2) = J'(n,l) - P(J'(n-l), 2 ; V(n)) (4) 
10 where the index (,2) in R(.) and P(.) now designates this "second" encoder 34, 

P(J'(n-l),2 ; V(n)) is the prediction which has to be subtracted from the output J'( n -0 of the 
logo adder 33 in order to obtain the residual signal that has to be coded, and R(n) designates 
said residual signal 

Finally, for the second decoder 35, one has (similarly to the previous case of the 
15 first decoder 32) : 

J'(n,2) = R'(n,2) + P(J'(n-l), 2 ; V(n)) (5) 
where the index (,2) in J'(.)> R '() and P(.) now designates this "second" decoder 35, R'(n) is 
the decoded residual signal, P(J*(n-l), 2 ; V(n)) is the prediction to be added to R'(n)» and 
J'(n) designates the output of the decoder. As for the decoder 32, J'(ii,2) may be also be 
20 written in the form : 

J'(n,2) = J , (n,l) + e(n,2) 

J'(n,2) = I'(n,l) + Logo(n) + e(n,2) 

J'(n,2) = I(n) + e(n,l ) + Logo(n) + e(n^) (6) 
which means that the output signal J'(ii,2) of the transmission chain is indeed equal to the sum 
25 of the original input signal I(n), of the first coding error (coding/decoding in the first encoder 
and decoder), of the second coding error (coding/decoding in the second encoder and 
decoder), and of the additional data. 

Then, using the linearity of the motion compensation operator, it may be 

written: 

30 P(r(n-1), 2 ; V(n)) = P[((J'(n-l), 1) + (e(n-l), 2)) ; V(n)] (7) 

which allows to write the relation (4) in the form : 

R(n,2) = J'(n,l) - P[((J'(n-l), 1) + (e(n-l), 2)) ; V(n)] (8) 
Using again the linearity of the compensation operator, one has : 

P(J'(n-l), 1 ; V(n)) = P[(I'(n-l) + Logo(n-l)), 1 ; V(n)] 
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or : 

P(J'(n-l), 1 ; V(n)) = P(T(n-l), 1 ; V(n)) + P(Logo(n-l), 1 ; V(n)) (9) 
Therefore the relation (8) becomes : 

R(n,2) - F(n,l) + Logo(n) - P(e(n-1),2 ; V(n)) 
5 - P(r(n-1), 1 ; V(n)) - P(Logo(n-l), 1 ; V(n)) (10) 

or, from the relation (2) : 

R(n,2) = R'(n-l) - P(e(n-1), 2 ; V(n)) 

+ Logo(n)-P(e(n-l),2;V(n)) 

- P(I'(n-I), 1 ; V(n)) - P(Logo(n-l), 1 ; V(n)) (11) 
1 0 which finally leads to the following relation (12) : 

R(n,2) = R'(n-l) - P(e(n-1), 2 ; V(n)) 

+ Logo(n) - P(Logo(n-l) ; V(n)) (12) 
which is the final system equation of the transcoder with logo adder as proposed according to 
the invention. 

15 The general outline of the transcoder without logo adder is first recalled in Fig.4 

(for comparison with Fig.5). It comprises a residue decoding branch 41 (variable length 
decoding VLD + inverse quantization IQ + inverse discrete cosine transform IDCT), an 
encoding and decoding branch 42 (discrete cosine transform DCT + quantization Q + variable 
length coding VLC ; inverse quantization IQ + inverse discrete cosine transform IDCT), and 

20 an intermediary branch called a pseudo-prediction branch 43 (first subtracter S + memory 

MEM + motion compensation COMP on the basis of motion vectors V(n) + second subtracter 
S). This branch 43 is so called because it is not exactly a classical prediction as in a basic 
encoder, the first adder being here replaced by a subtracter. The signals R'(n,l), R(n,2), 
R'(n,2), e(n,2), V(n), P(e(n-1), 2 ; V(n)) previously cited are shown in Fig.4. 

25 The corresponding scheme of the transcoder according to the invention -i.e. 

with a logo adder- is then illustrated in Fig.5 where, in comparison with Fig.4, the identical 
parts are designated in the same manner. The additional part is a logo addition branch 50 that 
includes a memory MEM 51, receiving the logo to be added (signal Logo(n)), a motion 
compensation circuit COMP 52, receiving the output of the memory 51 and the vectors V(n) 

30 and delivering a predicted data stream, a subtracter S 53, for delivering the difference between 
the original signal Logo(n) and the motion compensated one P(Logo(n-l) ; V(n)) available at 
the output of the circuit 52 (predicted data stream), and an adder 54, for introducing the output 
signal of said subtracter 53 into the main bitstream (corresponding to the complete sequence of 
successive pictures). The logo addition is therefore implemented by means of a residue 
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addition to the incoming bitstream, and this residue is formed by subtracting logo with a 
motion compensated logo prediction that is based on reference pictures containing logo 
previously stored and that uses the same modes and vectors as the main incoming bitstream. 

The method and device hereabove described may be further improved. It is 

5 indeed contemplated no longer to add, but to insert additional data (such as a logo) into the 
main video bitstream, without modifying the other parts of the picture. 

Such an insertion of data, i.e. the replacement of an area of a picture (or 
sequence of pictures) by said data- is a non-linear processing that requires a full access to the 
original pixels (= picture elements). For a high quality logo insertion, a full decoding of the 

10 incoming bitstream should therefore be performed. According to the invention, such a full 

decoding is replaced by a low-cost rough decoding (here described in the case of the MPEG-2 
standard). 

The corresponding scheme of the transcoder according to the invention -i.e. 
with a logo inserter - is illustrated in Fig.6 where, in comparison with Fig.4, the identical parts 

15. are designated in the same manner. The additional part (with respect to Fig.4) is a logo 
insertion device 400 that will be now described in a detailed manner. 

In this logo insertion device, four steps are implemented. The first one is a 
so-called DC estimation step, carried out in an estimation stage 410. Each picture of an MPEG 
sequence is subdivided into motion compensation units called macroblocks. Moreover, in 

20 MPEG standard, three types of pictures are considered : I (or intra) pictures, coded without 

any reference to other pictures, P (or predicted) pictures, coded with reference to a past picture 
(I or P), and B (or bidirectionally predicted) pictures, coded with reference to a past and a 
future picture (I or P). These I and P pictures will be called reference frames. In the stage 410, 
the average value of each macroblock is estimated according to the average values of the 

25 reference frames (previously stored in associated buffer memories 401 and 402) and to the 
current average value of the incoming residue (the signal referenced R' (n,l) in Fig.4). If 
DCP(b,Y) designates the average value of the prediction PR(b,Y) for the macroblock b in the 
component Y (an illustration of such a computation is given in Fig.7 in the case of an 
interpolated macroblock : the macroblock in each of the two reference pictures and the 

30 prediction PR(b,y) and its average value DCP(b,Y) are shown), if ACR(b,Y) designates the 
average value of the residue (i.e. of the signal called R*(n,l) at the output of the residue 
decoding branch 41 of the transcoder) for the same macroblock in the same component Y, and 
if MDC(b,Y) designates the estimated average value of the same macroblock b in the same 
component Y, then one has : 
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MDC(b,Y) = DCP(b, Y) + ACR(b,Y) (13) 
The description is given in the case of the component Y, but it would be the same with anyone 
of the chrominance components U,V in place of the luminance Y. 

It may be observed that no noticeable drift may occur on such estimations if it 
5 is assumed that the incoming bitstream, at the input of the transcoder, presents a standard 
broadcasting GOP duration (e.g. N = 12) allowing prediction pictures to be refreshed often 
enough on intra pictures (a GOP, or Group Of Pictures, consists of an I picture and all 
succeeding pictures until the next I picture, itself not included, and N is therefore defined as 
the size of the GQP, the most common one in MPEG standard being N = 12). 
10 The second step is a logo processing one, carried out in a processing stage 420. 

A processing of the logo pixel values is indeed performed in order to keep visibility and 
conformance of the logo whatever the picture content. It is in fact wanted to have : 

MV(CP) + MV(Logo(CP)) = RQM(CP) (14) 
where CP designates the luminance component Y or anyone of the chrominance components 
1 5 (U,V), MV(CP) is the average value of the original area where the logo has to be inserted in 
the component CP (these values are estimated according to the relation (13) indicated above), 
MV(Logo (CP)) is the average value of the processed logo, and RQM(CP) corresponds to the 
required average value of the picture after logo insertion, in the logo area. The pixel values 
Logo(b, CP) of the logo to be inserted at the macroblock b are therefore defined, after logo 
20 processing, by the relation (15) : 

Logo(b, CP) = Logo_ori(b, CP) + RQM(CP) 

- MV(CP) - MV_Logo_ori(CP) (15) 
where Logo__ori(b, CP) designate the original pixel values of the logo to be inserted at the 
macroblock b, and MV_Logo_ori(CP) designates the average value of the original logo in the 
25 component CP. 

This logo processing is illustrated in Figs.8 to 10 : Fig.8 shows the original logo 
signal Logo _pri(b, CP) before insertion, its average value MV Logo ori(CP), RQM(CP), and 
MV(CP), while Fig.9 shows what is required (MV(CP) + Logo(b, CP) = RQM(CP) and Fig. 10 
illustrates the principe of the adaptation carried out to obtain the required result (as indicated, a 
30 shift of the logo signal of a value equal to (MV(CP) - RQM(CP)) + MV_Logo_ori(CP)). 

For chrominance components, logo conformance to the original logo is 
required, i.e. the insertion of the processed logo into the picture must lead to the colors of the 
original logo. For the chrominance components U and V, RQM(CP) is therefore 
MV_Logo_ori(CP). For luminance, a choice can be made between either preferring 
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conformance to the original (i.e. RQM(Y) = MV_Logo__ori(Y)) or visibility (so that the logo 
appears dark in bright areas and vice-versa). In this last case, RQM(Y) = 128. 

The third step is a logo clipping one, carried out in a clipping stage 430. 
According to the MPEG-2 standard, at the decoder side the reconstructed pictures stored in 

5 memories are clipped between 0 and 255. To do the same at the encoder side cannot be 

achieved since a complete decoding of the incoming bitstream is not performed. The clipping 
step is then performed on a macroblock basis. The right pixel values are not known, but this 
situation is compensated by taking a margin MG on the clipping. Then, for each component Y, 
U or V (always called CP), Logo(b, CP) is clipped so that : 

10 0 + MG< MV(b, CP) + Logo(b, CP) < 255 - MG (16) 
which gives : 

MG - MV(b, CP) < Logo(b, CP) 0 7 ) 

Logo(b, CP) < 255 - MG - MV(b, CP) (18) 
This logo clipping is illustrated in Figs. 1 1 and 12, where Fig.l 1 allows 
1 5 to represent MV(b, CP), MV(b, CP) + Logo (b, CP), and the clipping limits MG and 255 - 

MG, and therefore illustrates the relation (16), while Fig. 12 illustrates similarly the relations 

(17) and (18). 

The fourth step is a logo addition one, carried out in a logo addition stage 440 
that is identical to the logo addition branch 50 shown in Fig.5. Said stage 440 comprises a 
20 memory receiving the clipped logo available at the output of the clipping stage 430, a motion 
compensation circuit receiving the output of said memory and the vectors V(n) defined in the 
transcoder, a subtracter (for delivering the difference between the clipped logo, available at the 
output of the preprocessing part of the logo insertion device 400, and the motion compensated 
one available at the output of said motion compensation circuit) and an adder (for inserting the 

25 output signal of said subtracter into the main bitstream). These four elements (memory, motion 
compensation circuit, subtracter, adder) are similar to the four elements 51 to 54 of Fig.5. The 
logo insertion is therefore implemented by means of a residue addition to the incoming 
bitstream, and this residue is formed by defining the difference between the logo 
(preprocessed as described) and a motion compensated prediction derived from said 

30 preprocessed logo and that uses the same motion vectors as the main incoming bitstream. 
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CLAIMS: 



10 



25 



1 . A method of modifying data in an encoded data stream corresponding to 

successive pictures divided into sub-pictures, comprising the steps of : 

- decoding said encoded data stream ; 

- re-encoding the decoded data stream ; 
characterized in that it also comprises the steps of : 

- defining a predicted data stream on the basis of an input modifying data 

stream ; 

- adding to the decoded data stream, before re-encoding it, the difference 
between the modifying data stream and the predicted one. 



2. A method according to claim 1, in which a pseudo-prediction sub-step is 

provided between said decoding and re-encoding steps, said sub-step comprising in series the 
following operations : 

- a definition operation of a coding error appeared during the re-encoding step ; 
15 - a motion compensation operation on the basis of motion vectors previously 

determined for each respective sub-picture and applied to said coding error ; 

- a subtracting operation defining a residual signal by difference between the 
decoded data stream and the output stream of the motion compensation operation ; 

said method being characterized in that said definition step comprises the operations of : 
20 - storing the modifying data stream ; 

- compensating motion on the basis of said motion vectors applied to said 
stored data stream ; 

- subtracting from the modifying data stream the predicted data stream available 
after said motion compensation operation. 



3. A video processing device for adding data to a video coded data stream, 

characterized in that it comprises : 

(1) a transcoder, for decoding said data stream and re-encoding it ; 

(2) an addition branch, for defining : 
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(i) on the basis for these additional data and motion vectors defined in 
the transcoder, a predicted signal ; 

(ii) the difference between said additional data and said predicted 
signal, in view of an addition of said difference into said data stream between decoding and re- 

5 encoding parts of said transcoder. 

4 # A method according to anyone of claims 1 and 2, characterized in that it also 

comprises the step of : 

- preprocessing the modifying data stream in order to obtain in the decoded data 
10 stream a substitution of said modifying data to the decoded ones. 

5. A method according to claim 4, in which a pseudo-prediction sub-step is 

provided between said decoding and re-encoding steps, said sub-step comprising in series the 
following operations : 

15 - a definition operation of a coding error appeared during the re-encoding step ; 

- a motion compensation operation on the basis of motion vectors previously 
determined and applied to said coding error ; 

- a subtracting operation defining a residual signal by difference between the 
decoded data stream and the output stream of the motion compensation operation ; 

20 said method being characterized in that said preprocessing step includes the operations of : 

- estimating the average values of all the sub-pictures of the current picture 
corresponding to said data stream ; 

- preprocessing the additional data in order to keep visibility and conformance 
of the additional data whatever the picture content is ; 

25 - clipping said preprocessed additional data on a macroblock basis. 

6. A video processing device for inserting data into a video coded data stream, 

characterized in that it comprises : 

(1) a transcoder for decoding said data stream and re-encoding it ; 
30 (2) an insertion branch, for defining : 

(i) on the basis of these additional data a preprocessed signal ; 

(ii) on the basis of said preprocessed signal and motion vectors defined 
in the transcoder, a predicted signal ; 
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(iii) the difference between said preprocessed additional data and 
predicted signal, in view of an insertion of said difference into said data stream between 
decoding and re-encoding parts of said transcoder. 
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